nlp-speech

Speech-to-Text Conversion

Techniques, Tools, and Trade-offs

Omkar Ninav
June 30, 2025

Good [morning/afternoon], thank you for taking the time to attend this presentation.

The goal today is to give a structured and technical overview of speech-to-text conversion — how it works, what tools and models are currently leading the field, and what trade-offs are involved in choosing a particular solution.

This is a technology that’s become foundational in a wide range of domains — from customer support and assistive tech, to real-time captioning and voice interfaces. I’ll be focusing on how Python-based tools like Whisper, Wav2Vec2, and others can be used to build robust STT systems.

By the end, I hope to outline clearly: - How the core pipeline functions, - Which models are best suited for various use cases, and - What the cost and accuracy implications are — especially in terms of practical deployment.

Let’s begin with the basics.

What is Speech-to-Text?

Converts spoken language into written text
Also known as Automatic Speech Recognition (ASR)
Used in applications like:
- Virtual assistants (e.g., Siri, Alexa)
- Captioning & subtitling
- Meeting transcription
- Voice commands

How Speech-to-Text Works

Audio Input
Voice signal captured from a microphone or file
Feature Extraction
Converts raw audio into numerical features (e.g., MFCCs, spectrograms)
Acoustic Model
Maps features to phonemes or characters using ML/DL models
Language Model
Predicts word sequences for context-aware transcription
Decoder
Aligns acoustic and language models to produce final text

flowchart LR
  A[Audio Input<br/>Voice signal] --> B[Feature Extraction<br/>MFCCs / Spectrograms]
  B --> C[Acoustic Model<br/>ML/DL-based]
  C --> D[Language Model<br/>Contextual Prediction]
  D --> E[Decoder<br/>Final Text Output]

This slide breaks down the standard architecture behind most modern speech-to-text systems.

It starts with the audio input, typically a microphone recording or audio file, which is passed to a feature extraction module. This module transforms the raw waveform into meaningful patterns — usually through methods like MFCCs (Mel-Frequency Cepstral Coefficients) or spectrograms.

Next, the acoustic model interprets these features and maps them to likely phonemes or character sequences. Traditionally, these were HMM-based, but today they’re often deep neural networks like CNNs, RNNs, or transformers.

The language model plays a critical role — it uses linguistic context to predict the most likely word sequences. This is what helps the system understand the difference between similar-sounding phrases like “ice cream” vs “I scream.”

Finally, the decoder combines all of this — the acoustic predictions and the language probabilities — to produce the final transcription.

Understanding this pipeline helps clarify where different models and libraries fit in, and what we can tweak based on accuracy, speed, or deployment constraints.

Techniques: Traditional ASR

HMMs + GMMs
Early models for mapping acoustic features to phonemes
n-gram Language Models
Predict word sequences based on prior word probabilities
Feature Extraction (MFCCs)
Converts raw audio into spectral features

Let’s start with traditional ASR systems.

Early speech recognition was built using Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs). HMMs model the sequential nature of speech, and GMMs handle the probability distributions of acoustic features within each state.

For predicting the next word or phrase, systems used n-gram language models — for example, a trigram model would look at the two previous words to estimate the next one. These models were easy to implement but had limited ability to understand context or long-term dependencies.

The front end of these systems used MFCCs (Mel-Frequency Cepstral Coefficients) to convert audio waveforms into feature vectors that captured the shape and tone of the speech signal. MFCCs remain popular even in some deep learning systems due to their compact and informative representation of sound.

While these traditional methods laid the foundation for modern ASR, they struggled with complex variations like accents, overlapping speakers, and noisy environments.

Techniques: Deep Learning & Modern Trends

RNNs, CNNs, Transformers
Deep models for sequence modeling and attention
End-to-End Models (CTC / Attention)
Learn transcription directly from audio input
Trends
- Self-supervised learning (e.g. Wav2Vec 2.0)
- Multilingual support (e.g. Whisper)
- On-device inference (edge devices, mobile)

Modern speech recognition systems have shifted toward deep learning.

Initially, Recurrent Neural Networks (RNNs) were used because they could model sequences — perfect for speech. Then came Convolutional Neural Networks (CNNs) to handle time-frequency patterns in audio. Today, transformers are dominant due to their superior performance in modeling long-range dependencies and context.

Instead of breaking the process into multiple modules like traditional ASR, deep learning systems often use end-to-end models. These can be trained to map directly from raw audio to text using techniques like Connectionist Temporal Classification (CTC) or attention-based models.

Recent trends in STT are pushing the field even further: - Self-supervised learning enables models like Wav2Vec 2.0 to learn from raw, unlabeled audio — drastically reducing the need for costly transcriptions. - Models like Whisper offer multilingual support and are robust to accents and background noise. - There’s also growing interest in on-device inference, where models are small and efficient enough to run on phones, edge devices, or embedded systems — ensuring both privacy and speed.

These modern techniques make STT more accurate, scalable, and accessible than ever before.

Python Libraries for Speech-to-Text (1/2)

Whisper (OpenAI)
- Transformer-based, multilingual
- High accuracy, runs offline
SpeechRecognition
- Simple API wrapper for Google, IBM, etc.
- Easy for beginners
Wav2Vec 2.0 (Hugging Face)
- Pretrained self-supervised models
- High-quality transcriptions

Python Libraries for Speech-to-Text (2/2)

DeepSpeech (Mozilla)
- Lightweight and fast
- Less accurate on noisy inputs
Kaldi (via PyKaldi)
- Research-grade toolkit
- Steeper learning curve
Vosk
- Real-time offline STT
- Works on Raspberry Pi, Android, desktops
- Multilingual, very lightweight

DeepSpeech, from Mozilla, was designed for simplicity and edge deployment. It’s fast and easy to install, but less accurate in noisy or real-world conditions compared to newer models.

Kaldi is a highly flexible and powerful toolkit often used in academia or enterprise. It offers full control over the speech recognition pipeline, including acoustic and language models. However, it’s not beginner-friendly and requires significant setup effort. The Python wrapper PyKaldi helps but still demands technical depth.

Vosk is an often-overlooked gem. It supports real-time offline transcription in 20+ languages, including Hindi, Marathi, and more. It runs even on low-resource devices like Raspberry Pi or Android phones. It’s ideal for small, local deployments where Whisper would be too heavy or cloud is not an option.

Splitting the content across two slides avoids overcrowding and gives us room to discuss each tool more clearly.

Model Comparison: Features at a Glance

Model	Accuracy	Offline	Multilingual	Ease of Use	Cost
Whisper	✅✅✅	✅	✅✅✅	✅✅	Free
Wav2Vec 2.0	✅✅	✅	⚠️ (mostly English)	✅	Free
Google API	✅✅✅	❌	✅✅✅	✅✅✅	Paid
DeepSpeech	✅	✅	❌	✅✅	Free
Kaldi	✅✅	✅	✅ (with effort)	⚠️ Complex	Free
Vosk	✅✅	✅	✅✅	✅✅✅	Free

This table summarizes the strengths and limitations of some of the most widely used speech-to-text models and toolkits.

Starting with Whisper, it stands out as a balanced option — it offers high accuracy, works offline, supports multiple languages, and is open-source. It does require a decent GPU to run efficiently, especially for long audio files or real-time use, but it’s very accessible for developers and teams with modest resources.

Wav2Vec 2.0 is another excellent option. It provides high accuracy, especially when fine-tuned on domain-specific data. However, it’s primarily trained on English datasets, so multilingual support is more limited unless you retrain it. It’s open-source and can be accessed via Hugging Face.

Google Speech-to-Text API is a commercial solution. It offers very high accuracy and supports over 100 languages. It’s excellent for real-time use cases and scales easily. However, it’s not free — the cost is around $1.44 per hour, and it requires internet connectivity, which may raise privacy concerns in sensitive environments.

DeepSpeech was designed to be lightweight and fast, making it suitable for edge devices. That said, it has started to lag behind in terms of accuracy, especially on noisy or accented inputs, and its development has slowed down in recent years.

Kaldi is extremely powerful and flexible. It’s used in many academic and enterprise applications where low-level control is needed — for instance, custom language models, speaker adaptation, and complex pipelines. However, it’s not beginner-friendly, and the learning curve is steep. PyKaldi can help if you want to work with it in Python.

Vosk is surprisingly effective given its small size. It offers decent accuracy, real-time performance, works fully offline, and is extremely easy to integrate — especially on embedded systems or mobile platforms.For projects targeting Indian languages or edge devices, Vosk can be a better fit than Whisper or cloud APIs.

This slide helps us decide which model might be most appropriate depending on our project constraints — whether we prioritize cost, offline access, multilingual support, or ease of use.

In the next slide, we’ll break down the cost aspect more specifically, including free tiers and usage pricing for cloud APIs.

Cost: Open-Source Models

Model	Cost	Offline Use	Cloud Required
Whisper	Free (Open Source)	✅	❌
Wav2Vec 2.0	Free (Open Source)	✅	❌
DeepSpeech	Free (Open Source)	✅	❌
Kaldi	Free (Open Source)	✅	❌
Vosk	Free (Open Source)	✅	❌

Cost: Cloud APIs

Service	Approx. Cost	Offline Use	Cloud Required
Google Speech API	~$1.44 per hour	❌	✅
AWS Transcribe	~$1.44 per hour	❌	✅
Azure Speech	~$1.60 per hour	❌	✅

Cost – Summary & Recommendation

Open-Source Models (Whisper, Wav2Vec, etc.)
- ✅ Free and offline
- ⚠️ Require setup and local resources
Cloud APIs (Google, AWS, Azure)
- ✅ Easy to use, scalable
- ⚠️ Ongoing cost and privacy trade-offs

✔️ Recommendation:
Use Whisper or Wav2Vec 2.0 for local, cost-effective transcription
Use Vosk for light, multilingual offline STT (e.g. edge devices)
Use Cloud APIs only for real-time or highly multilingual needs

✅ Final Decision

Chosen Model: openai/whisper-medium

🔹 Why Whisper Medium?

✅ Good balance between accuracy and inference speed
✅ Lower VRAM & compute requirements than large models
✅ Suitable for local deployment (runs on RTX 4050)
✅ Consistent performance across test cases
✅ Robust to mild accents and moderate background noise

🔸 Not Chosen:

whisper-large: Higher accuracy but heavier (more VRAM)
whisper-tiny / base: Faster, but less accurate

📝 Transcript Evaluation – Whisper vs Original

🎙️ Sample: NPTEL Lecture – Literature History

📌 Original:
> At the outset, this being the first session, it is very important to give an overview of the course. This course is spread over 12 weeks and we may have 30 hours of teaching involved in this. Let me also introduce you to the objectives of this course so that the intentions become clearer.

🌀 Whisper Output:
> At the outset, this being the first session, it is very important to give an overview of the course. This course is spread over 12 weeks and we may have 30 hours of teaching involved in this and we also introduce you to the objectives of this course so that the intentions become more clear.

🔹 ✅ Core content retained
🔹 ⚠️ Minor stylistic variation – rephrasing & joined sentences

🧠 Observation Summary

Whisper captured all key points with high fidelity
Punctuation differences due to lack of postprocessing
Filler words (“we also”) inserted naturally from speech
Ideal for lecture summarization, accessibility, or note generation

📊 Whisper Accuracy

🔹 Quantitative Evaluation

Semantic Similarity (cosine): 0.9766 ✅
ROUGE-1 F1 Score: 0.9714
ROUGE-L F1 Score: 0.9524
Word Error Rate (WER): 12.43%

🔍 Interpretation

✅ Very high semantic match – meaning fully preserved
✅ Low WER – minor word-level issues
✅ ROUGE scores confirm strong overlap in phrase structure
⚠️ Differences mostly in punctuation, filler words, and phrasing

📝 Transcript Comparison – Whisper vs Original

🎙️ Sample: NPTEL Lecture – Taylor Series Intro

📌 Original:
> Welcome back to the lectures on Engineering Mathematics-I.
> Today, we will learn Taylor’s Polynomial and Taylor Series.

🌀 Whisper Output:
> Hi, welcome back to the lectures on Engineering Mathematics I and today’s we will learn Taylor’s polynomial and Taylor series.

🔹 ✅ Core message preserved
🔹 Minor changes:
- Added “Hi”
- “today’s we” instead of “today we”
- Punctuation differences only

🎙️ Sample: Exponential Function Example

📌 Original:
> The polynomial of degree 0 will simply be 1.
> If we plot this, it’s the green line through the point (0, 1).

🌀 Whisper Output:
> So, the polynomial of degree 0 will be simply 1 and if we plot this. So, this is the green plot here of the exponential function and this polynomial of degree 0 is just a constant line. So, the straight line going through this 0 1 point.

🔹 ✅ Richer phrasing from audio
🔹 Whisper added spontaneous repetitions and fillers (“so”, “here”)
🔹 All technical meaning retained

📊 Whisper Accuracy

🔹 Quantitative Evaluation

Semantic Similarity (cosine): 0.8882
ROUGE-1 F1 Score: 0.9116
ROUGE-L F1 Score: 0.8913
Word Error Rate (WER): 26.33%

🔍 Interpretation

✅ Meaning mostly preserved, despite phrasing variation
⚠️ Slight drop in surface-level overlap (function words, rewording)
⚠️ Higher WER due to longer output and added filler words

Conclusion & Summary

STT is a mature, versatile technology
Open-source tools (Whisper, Wav2Vec) offer high quality with no cost
Cloud APIs provide convenience but incur recurring costs
Model choice depends on:
✅ Accuracy
✅ Cost
✅ Deployment constraints
✅ Privacy needs

✔️ Recommended:
Use Whisper for secure, offline transcription
Use cloud APIs only where real-time & scalability are critical

Thank You!

Questions? Suggestions?

🔗 View this presentation on GitHub:

Presented by Omkar Ninav — June 2025